Information processing device, information processing method, and information processing program

ABSTRACT

An information processing device ( 100 ) according to the present disclosure includes: an acquisition unit ( 141 ) configured to acquire a first image including a content image of an ear of a user; and a calculation unit ( 142 ) configured to calculate, based on the first image acquired by the acquisition unit ( 141 ), a head-related transfer function corresponding to the user by using a learned model having learned to output a head-related transfer function corresponding to an ear when an image including a content image of the ear is input.

FIELD

The present disclosure relates to an information processing device, aninformation processing method, and an information processing program.Specifically, the present disclosure relates to a head-related transferfunction calculation processing.

BACKGROUND

A technology of stereoscopically reproducing an audio image at aheadphone or the like by using a head-related transfer function(hereinafter also referred to as an HRTF) that mathematically expresseshow sound travels from a sound source to an ear has been used.

The head-related transfer function largely differs among individuals,and thus it is desirable to use a head-related transfer functiongenerated for each individual when the head-related transfer function isused. For example, a technology of generating a three-dimensionaldigital model (hereinafter referred to as a “3D model”) of a head basedon an image obtained by capturing an auricle of a user, and calculatinga head-related transfer function of the user based on the generated 3Dmodel has been known.

CITATION LIST Patent Literature

Patent Literature 1: U.S. Pat. No. 9,544,706

SUMMARY Technical Problem

According to a conventional technology, it is possible to use, ininformation processing, a head-related transfer function individuallycalculated in accordance with each user, thereby increasing stationaryof an audio image.

However, the above-described conventional technology generates a 3Ddigital model based on an image captured by a user and calculates ahead-related transfer function based on the generated model, and thushas a relatively large calculation processing load. Thus, it is presumedthat, with the above-described conventional technology, a long time isneeded until a head-related transfer function is provided to a userhaving transmitted an image, and accordingly, convenience is not high.

Thus, the present disclosure provides an information processing device,an information processing method, and an information processing programthat can improve user convenience in processing related to ahead-related transfer function.

Solution to Problem

To solve the above problem, an information processing device accordingto an embodiment of the present disclosure includes: an acquisition unitconfigured to acquire a first image including a content image of an earof a user; and a calculation unit configured to calculate, based on thefirst image acquired by the acquisition unit, a head-related transferfunction corresponding to the user by using a learned model havinglearned to output a head-related transfer function corresponding to anear when an image including a content image of the ear is input.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating an overview of information processingaccording to a first embodiment of the present disclosure.

FIG. 2 is a conceptual diagram illustrating the entire process ofinformation processing according to the present disclosure.

FIG. 3 is a diagram illustrating an exemplary configuration of aninformation processing device according to the first embodiment of thepresent disclosure.

FIG. 4 is a diagram illustrating an exemplary ear image storage unitaccording to the present disclosure.

FIG. 5 is a diagram illustrating exemplary learning processing relatedto an ear model according to the present disclosure.

FIG. 6 is a diagram illustrating exemplary learning processing relatedto an ear parameter estimation model according to the presentdisclosure.

FIG. 7 is a diagram illustrating exemplary ear image generationprocessing according to the present disclosure.

FIG. 8 is a diagram for describing the ear parameter estimation modelaccording to the present disclosure.

FIG. 9 is a diagram illustrating an outline of the process of processingof generating a model related to an HRTF.

FIG. 10 is a diagram for describing 3D model reconstruction according tothe present disclosure.

FIG. 11 is a diagram for describing details of the processing ofgenerating the model related to the HRTF.

FIG. 12 is a diagram for describing compression and decompression of theHRTF according to the present disclosure.

FIG. 13 is a diagram illustrating the process of HRTF estimationprocessing according to the present disclosure.

FIG. 14 is a diagram illustrating an exemplary configuration of a userterminal according to the first embodiment of the present disclosure.

FIG. 15 is a diagram illustrating the process of detection processingaccording to the present disclosure.

FIG. 16 is a diagram illustrating an exemplary configuration of aninformation processing device according to a second embodiment of thepresent disclosure.

FIG. 17 is a hardware configuration diagram illustrating an exemplarycomputer configured to achieve functions of each information processingdevice.

DESCRIPTION OF EMBODIMENTS

Embodiments of the present disclosure will be described below in detailwith reference to the accompanying drawings. Note that identical sitesin the embodiments are denoted by an identical reference sign, andduplicate description thereof is omitted.

1. First Embodiment [1-1. Outline of Information Processing According toFirst Embodiment]

First, the configuration of an information processing system 1 accordingto the present disclosure and an outline of information processingexecuted by the information processing system 1 will be described withreference to FIG. 1. FIG. 1 is a diagram illustrating an outline ofinformation processing according to a first embodiment of the presentdisclosure. The information processing according to the first embodimentof the present disclosure is achieved by the information processingsystem 1 illustrated in FIG. 1. The information processing system 1includes an information processing device 100 and a user terminal 10.The information processing device 100 and the user terminal 10 performmutual communication through a wired or wireless network (notillustrated). Note that the number of devices of each kind included inthe information processing system 1 is not limited to that illustrated.

The information processing device 100 is an exemplary informationprocessing device according to the present disclosure calculates ahead-related transfer function (HRTF) corresponding to each user andprovides the calculated HRTF. The information processing device 100 isachieved by, for example, a server device.

The user terminal 10 is an information processing terminal used by auser desired to receive HRTF provision. The user terminal 10 is achievedby, for example, a smartphone having an image capturing function. In theexample illustrated in FIG. 1, the user terminal 10 is used by a userU01 as an exemplary user.

An HRTF expresses, as a transfer function, sound change caused by anearby object including the shape of an auricle (ear conch) of or thehead of a person. Typically, measurement data for calculating an HRTF isacquired by measuring an acoustic signal for measurement by using amicrophone mounted in an auricle of a person, a dummy head microphone,or the like.

For example, an HRTF used in a technology such as a 3D acoustictechnology is often calculated by using measurement data acquired by adummy head microphone or the like, an average value of measurement dataacquired from a large number of people, or the like. However, the HRTFsof individuals largely different from each other, and thus it isdesirable to use the own HRTF of a user to achieve a more effectiveacoustic rending effect. Specifically, it is possible to provideacoustic experience with a more realistic sensation to a user byreplacing a typical HRTF with the HRTF of the user.

However, measurement of the HRTF of an individual user has various kindsof problems. For example, relatively highly dense measurement data isneeded to obtain an HRTF that provides an excellent acoustic effect.Measurement data of acoustic signals output to a user at various anglesaround the user is needed to acquire highly dense measurement data. Suchmeasurement needs a long time and thus a physical burden on the user islarge. In addition, accurate measurement needs to be performed in ananechoic room or the like, and thus an economic burden is large as well.Thus, for example, reduction of a burden on a user and reduction ofmeasurement cost are problems in HRTF calculation.

As for the above-described problems, there is a technology of expressingan ear or the head of a user in a 3D model and performing pseudomeasurement through acoustic simulation with the 3D model. With thetechnology, the user can perform calculation of an HRTF by providingscanning data of the head or a captured image of the head withoutperforming actual measurement in a measurement room.

However, a calculation processing load is extremely large for processingof generating a 3D model and acoustic simulation with the 3D model.Thus, a time lag of several ten minutes or several hours potentiallyoccurs when the above-described technology is used to incorporate anHRTF unique to a user on software or the like that uses, for example, 3Dacoustic. This is not highly convenient for the user. Accordingly,processing speed in HRTF calculation needs to be improved to achieveeffective use of an HRTF by a user, which is another problem.

As described above, there are various kinds of problems with obtainingan HRTF corresponding to an individual user. The information processingdevice 100 according to the present disclosure solves theabove-described problems through the information processing of thepresent disclosure.

Specifically, the information processing device 100 calculates an HRTFcorresponding to a user by using a learned model (hereinafter simplyreferred to as a “model”) having learned to output an HRTF correspondingto an ear when an image including a content image of the ear is input.For example, when having acquired an image including a content image ofan ear of the user U01 from the user terminal 10, the informationprocessing device 100 calculates an HRTF unique to the user U01 byinputting the image into the model. Specifically, the informationprocessing device 100 calculates an HRTF not through the process ofgenerating a 3D model based on the image of the user U01 and performingacoustic simulation.

Accordingly, the information processing device 100 can perform HRTFcalculation in an extremely short time as compared to a case in whichacoustic simulation is executed. The outline of the informationprocessing executed by the information processing device 100 accordingto the present disclosure will be described below with reference to FIG.1 along the process thereof.

As illustrated in FIG. 1, the user U01 performs image capturing of theuser from a side of the head to acquire an image including a contentimage of an ear of the user, (step S1). For example, the user U01performs image capturing of the head of the user by using a camera ownedby the user terminal 10. Note that in the present disclosure, an imageof an ear is not limited to a typical two-dimensional color image thatcan be captured by the user terminal 10 or the like but may be amonochrome image, a depth image including depth information, or anoptional combination thereof. The number of images used in theprocessing of the present disclosure is not limited to one but may betwo or more.

The user terminal 10 executes, on an image 5 obtained at step S1,preprocessing for transmission to the information processing device 100(step S2). Specifically, the user terminal 10 performs, as thepreprocessing, processing of detecting a content image of an ear of theuser U01 included in the image 5 and cutting out the range of thedetected ear content image from the image 5. Note that details of thepreprocessing such as ear detection, will be described later.

Through the preprocessing, the user terminal 10 generates an image 6including a content image of an ear of the user U01. Then, the userterminal 10 transmits the image 6 to the information processing device100 (step S3). Note that a series of pieces of processing such as theprocessing of producing the image 6 from the image 5 obtained by imagecapturing and the processing of transmitting the image 6 are executedby, for example, a computer program (for example, a smartphoneapplication) provided by the information processing device 100. In thismanner, in the information processing according to the presentdisclosure, only the image 6 obtained by cutting out only the earcontent image from the image 5 is transmitted to a network, but theimage 5 with which the user is potentially specified is not transmitted,and thus processing that is excellent in security can be achieved. Inaddition, the information processing device 100 does not acquire theimage 5 but acquires only the image 6, thereby avoiding the risk ofhandling private information. Note that the information processingdevice 100 may perform processing of acquiring the image 5 from the userterminal 10 and producing the image 6 on the information processingdevice 100 side. This configuration will be described later as a secondembodiment.

The information processing device 100 acquires the image 6 transmittedfrom the user terminal 10. Then, the information processing device 100inputs the image 6 to a model stored in a storage unit 120 (step S4).This model is a model having learned to output an HRTF corresponding toan ear when a two-dimensional image including a content image of the earis input. Specifically, the information processing device 100 calculatesan HRTF corresponding to the ear (in other words, the user U01) includedin the image 6 by inputting the image 6 to the model.

Then, the information processing device 100 provides the calculated HRTFto the user terminal 10 (step S5). In this manner, the user U01 canobtain an HRTF unique to the user without actual measurement processing,3D model acoustic simulation, nor the like by only preparing the image 5through image capturing of a side face of the user. Accordingly, theinformation processing device 100 can provide an HRTF to the user U01 inan extremely short time without causing a measurement burden on the userU01. As a result, the information processing device 100 can achieveimproved convenience for a user in processing related to an HRTF.

As described above, in the information processing according to thepresent disclosure, an HRTF is calculated by using a model generatedthrough learning processing, thereby achieving speeding-up of theprocessing. Although the outline of the processing of providing an HRTFto the user U01 in the information processing according to the presentdisclosure is described with reference to FIG. 1, a series ofinformation processes performed by the information processing device100, which include model learning processing will be described below indetail with reference to FIG. 2 and the following diagrams. Note that a“model” described with reference to FIG. 1 is not necessarily one modelbut may be a combination of a plurality of models that output variouskinds of values, details of which will be described with reference toFIG. 2 and the following diagrams.

[1-2. Entire Process of Information Processing According to the PresentDisclosure]

Before describing details of such as the configuration of theinformation processing device 100, the entire process of the informationprocessing executed by the information processing device 100 accordingto the present disclosure is illustrated in FIG. 2. FIG. 2 is aconceptual diagram illustrating the entire process of the informationprocessing according to the present disclosure.

First, the information processing device 100 collects data related toear forms of a plurality of persons and generates an ear model based onthe collected ear forms (step S11). Note that an ear form is notnecessarily limited to plaster or the like shaped in an ear of a personbut may be any information indicating the shape of an ear of a person.In the present disclosure, an ear model is a model that outputs acorresponding ear shape when a parameter (hereinafter referred to as an“ear parameter”) indicating a characteristic of an ear is input. An earparameter can be obtained by, for example, performing principalcomponent analysis on the shape of an ear based on data (for example,data obtained from a collected ear form by computed tomography (CT)scanning) related to the shape of the ear. Accordingly, once an earparameter is obtained, the information processing device 100 can obtaindata of the shape of an ear corresponding to the ear parameter (in otherwords, a 3D model representing the ear).

Thereafter, the information processing device 100 generates an earparameter estimation model based on the ear model (step S12). Theinformation processing device 100 can generate a large number of earimages by inputting ear parameters to the ear model generated at stepS11. An ear parameter may be input at random, or an ear parameter may beautomatically generated based on an optional rule (for example, when thefact that particular tendency in the ear shape exists for eachparticular race is found, a rule may be derived based on this fact), andthe generated value may be input. Thus, the information processingdevice 100 can generate a model that outputs an ear parametercorresponding to an ear when an image including the ear is input bylearning the relation between each generated ear image and the earparameter based on which generation is performed. This model is an earparameter estimation model. Accordingly, once a two-dimensional imageincluding a content image of an ear is obtained, the informationprocessing device 100 can obtain an ear parameter corresponding to theear. Then, once the ear parameter is obtained, the informationprocessing device 100 can obtain a 3D model of the ear included in theimage by using the ear model generated at step S11. Note that, in theabove-described learning, the relation between an image obtained bycapturing an ear of a person, the shape of the ear being represented bydata, and an ear parameter converted from the data of the ear may belearned. In this case, learning using an actual captured image, not acomputer graphics (CG) image, is performed, and thus it is assumed thatthe accuracy of a generated ear parameter estimation model can beincreased.

The information processing device 100 performs acoustic simulation forthe 3D model generated by using the ear parameter estimation model andcalculates a unique HRTF (hereinafter, such an HRTF generated for anindividual ear form is referred to as a “personalized HRTF”)corresponding to the 3D model (step S13). Accordingly, through theprocesses at steps S11 to S13, the information processing device 100 canachieve a series of pieces of processing for calculating a personalizedHRTF from an image including an ear by performing acoustic simulation.

In addition, the information processing device 100 generates a largenumber of 3D models based on randomly or regularly generated earparameters and repeats processing of performing acoustic simulation forthe generated 3D models, thereby learning the relation between an earparameter and a personalized HRTF. Accordingly, the informationprocessing device 100 generates an HRTF learning model based on thecalculated personalized HRTF (step S14).

In the present disclosure, an HRTF learning model is a model thatoutputs, when an ear parameter is input, a personalized HRTFcorresponding to the ear parameter. Accordingly, once an ear parameteris obtained, the information processing device 100 can obtain apersonalized HRTF corresponding to the ear parameter.

Thereafter, when having acquired an image from the user, the informationprocessing device 100 calculates the personalized HRTF of the user byinputting the image (more accurately, an ear parameter of an earincluded in the image) to the HRTF learning model (step S15). Theprocessing indicated at step S15 corresponds to the series of pieces ofprocessing illustrated in FIG. 1.

As described above, the information processing device 100 generates aplurality of models and performs information processing using thegenerated models, thereby calculating a personalized HRTF based on animage acquired from a user. Note that the pieces of processingillustrated in FIG. 2 do not necessarily need to be executed in theorder of steps S11 to S15. For example, the information processingdevice 100 only needs to perform the processing at step S15 aftergenerating each model necessary for processing.

This concludes the description of the outline of the entire process ofthe information processing according to the present disclosure. Theconfigurations of the information processing device 100 and the userterminal 10 as well as details of various kinds of learning processingwill be sequentially described with reference to FIG. 3 and thefollowing diagrams.

[1-3. Configuration of Information Processing Device According to FirstEmbodiment]

The configuration of the information processing device 100 according tothe first embodiment will be described below with reference to FIG. 3.FIG. 3 is a diagram illustrating an exemplary configuration of theinformation processing device 100 according to the first embodiment ofthe present disclosure.

As illustrated in FIG. 3, the information processing device 100 includesa communication unit 110, the storage unit 120, and a control unit 130.Note that the information processing device 100 may include an inputunit (for example, a keyboard or a mouse) configured to receive variousoperations from an administrator or the like who manages the informationprocessing device 100, and a display unit (for example, a liquid crystaldisplay) for displaying various kinds of information.

The communication unit 110 is achieved by, for example, a networkinterface card (NIC). The communication unit 110 is connected with anetwork N (such as the Internet) in a wired or wireless manner andtransmits and receives information to and from the user terminal 10 orthe like through the network N.

The storage unit 120 is achieved by a storage device, for example, asemiconductor memory element such as a random access memory (RAM) or aflash memory, a hard disk, or an optical disk. The storage unit 120stores various kinds of data used for learning processing, and a modelor the like generated by learning processing.

As illustrated in FIG. 3, the storage unit 120 includes an ear forminformation storage unit 121, an ear model storage unit 122, an earimage storage unit 123, an ear parameter estimation model storage unit124, an HRTF processing model storage unit 125, an HRTF learning datastorage unit 126, and an HRTF learning model storage unit 127.

The ear form information storage unit 121 stores information of 3D modeldata generated from an ear form actually collected from a human body (inother words, information related to the shape of an ear). Specifically,the ear form information storage unit 121 stores data (such as a 3Dpolygon) representing a three-dimensional shape and obtained byperforming CT scanning of the collected ear form.

The ear model storage unit 122 stores an ear model according to thepresent disclosure. An ear model is a model that outputs, when an earparameter indicating a characteristic of an ear is input, acorresponding ear shape.

An ear parameter can be obtained by performing principal componentanalysis on data indicating the shape of an ear form stored in the earform information storage unit 121. Specifically, an ear parameter isobtained by quantifying a site that largely changes in an ear(characterizes the shape of the ear) by performing statistical analysis(principal component analysis) on a 3D polygon of the ear. An earparameter according to the present disclosure is indicated as, forexample, a combination of 10 numbers, and each number is indicated as,for example, a number in the range of minus 10 to plus 10. For example,an ear parameter for which all numbers are “0” corresponds to an earhaving an average shape of learning data (collected ear forms). Notethat the information processing device 100 may apply, as appropriate, aknown technology used in, for example, processing of generating the faceof a person to the processing of generating a model indicating the shapeof an ear by principal component analysis. The information processingdevice 100 is not limited to principal component analysis but maygenerate an ear parameter by using a known analysis method such asindependent component analysis or any other non-linear model asappropriate. An ear parameter is not limited to that obtainedquantifying a site that largely changes in an ear but may be, forexample, that obtained by parameterizing a characteristic related to theshape of an ear so that influence on an HRTF is large.

The ear image storage unit 123 stores an image including a content imageof an ear. For example, the ear image storage unit 123 stores, as an earimage, a CG image obtained by rendering the shape of an ear (3D model ofthe ear) generated by an ear model. The ear image storage unit 123 maystore, as an ear image, an image including a content image of an ear,which is transmitted from the user.

FIG. 4 illustrates an exemplary ear image storage unit 123 according tothe present disclosure. FIG. 4 is a diagram illustrating an exemplaryear image storage unit 123 of the present disclosure. In the exampleillustrated in FIG. 4, the ear image storage unit 123 has the items of“ear parameter”, “ear 3D model data”, “head 3D model data”, “ear imageID”, and “image generation parameter”. The “image generation parameter”has the sub-items of “texture”, “camera angle”, “resolution”, and“luminance”.

The “ear parameter” is a parameter indicating a characteristic of theshape of an ear. For example, the ear parameter is expressed by aten-dimensional number. The “ear 3D model data” is data indicating thethree-dimensional shape of an ear reconstructed based on an earparameter. The “head 3D model data” is data indicating thethree-dimensional shape of a head and synthesized with the ear 3D modeldata at reconstruction of a 3D model of a person.

The “ear image ID” indicates identification information that identifiesan ear image obtained by rendering a 3D model. As illustrated in FIG. 4,a plurality of ear images are generated from one 3D model by changing,in various manners, a parameter (image generation parameter) set atrendering.

The “image generation parameter” indicates a set parameter in renderingfor generating an image. The “texture” indicates a setting of CGtexture. The “camera angle” indicates the image capturing angle of apseudo camera when a two-dimensional image is obtained by rendering a 3Dmodel. The “resolution” indicates the resolution of rendering. The“luminance” indicates the luminance of rendering. The item of luminancemay include setting data such as the angle of light (incident light) atrendering.

Note that although data of each item is conceptually written as, forexample, “A01” or “B01” in FIG. 4, specific data corresponding to theitem is stored as data of the item in reality. For example, a specificstring of 10 numbers is stored in the item of “ear parameter”. This issame for the other items, and various numerical values and kinds ofinformation corresponding to each item are stored in each item.

Specifically, in the example illustrated in FIG. 4, it is indicated thatear 3D model data generated by ear parameter “A01” is “B01” and head 3Dmodel data combined with this ear 3D model data and included in a 3Dmodel of a person is “C01”. In addition, it is indicated that ear imagesobtained from a generated 3D model of a person are a plurality of earimages identified by ear images ID “D01”, “D02”, “D03”, and the like. Inaddition, it is indicated that, for an ear image identified by ear imageID “D01”, the texture is “E01”, the camera angle is “F01”, theresolution is “G01”, and the luminance is “H01” as image generationparameters at rendering.

Description continues with reference to FIG. 3 again. The ear parameterestimation model storage unit 124 stores an ear parameter estimationmodel. An ear parameter estimation model is a model that outputs, when atwo-dimensional image including a content image of an ear is input, anear parameter corresponding to the ear.

The HRTF processing model storage unit 125 stores an HRTF processingmodel. Although described later in detail, an HRTF processing modelperforms processing of compressing the amount of information of an HRTFcalculated by acoustic simulation or the like. Note that in thefollowing description, an HRTF compressed by an HRTF processing model isalso referred to as an HRTF parameter.

The HRTF learning data storage unit 126 stores learning data forgenerating a model (HRTF learning model to be described later) forcalculating an HRTF from an image including a content image of an ear.Specifically, the HRTF learning data storage unit 126 stores, aslearning data, data in which an ear parameter indicating the shape of anear and an HRTF corresponding to the shape of the ear specified based onthe ear parameter are combined.

The HRTF learning model storage unit 127 stores an HRTF learning model.An HRTF learning model is a model that outputs, when an image includinga content image of an ear is input, an HRTF corresponding to the ear.For example, when an image including a content image of an ear isacquired, an HRTF learning model uses an ear parameter estimation modelto output an ear parameter corresponding to the ear, and in addition,outputs an HRTF corresponding to the ear parameter.

The control unit 130 is achieved by, for example, a central processingunit (CPU) or a micro processing unit (MPU) executing, by using a randomaccess memory (RAM) or the like as a work area, a computer program (forexample, an information processing program according to the presentdisclosure) stored in the information processing device 100. The controlunit 130 is a controller and may be achieved by, for example, anintegrated circuit such as an application specific integrated circuit(ASIC) or a field programmable gate array (FPGA).

As illustrated in FIG. 3, the control unit 130 includes a learning unit131 and an estimation unit 140. The learning unit 131 includes areception unit 132, an ear model learning unit 133, an image generationunit 134, an ear parameter learning unit 135, and an HRTF learning unit136 and achieves or executes functions and effects of informationprocessing described below. The estimation unit 140 includes anacquisition unit 141, a calculation unit 142, and a provision unit 143and achieves or executes the functions and effects of the informationprocessing described below. Note that the internal configuration of thecontrol unit 130 is not limited to the configuration illustrated in FIG.3 but may be any configuration with which the information processing tobe described later is performed.

The learning unit 131 will be first described below. The learning unit131 performs learning processing related to various kinds of data andgenerates various models to be used by the estimation unit 140.

Note that the learning unit 131 performs learning for generating a modelbased on various kinds of data, but learning processing described belowis exemplary, and the type of learning processing executed by thelearning unit 131 is not specified to any type. For example, thelearning unit 131 may generate a model by using various learningalgorithms such as a neural network, a support vector machine,clustering, and reinforcement learning.

The reception unit 132 receives various kinds of information. Forexample, the reception unit 132 receives CT scanning data of an ear formcollected from a human body. The reception unit 132 stores the receiveddata in the ear form information storage unit 121.

The ear model learning unit 133 generates an ear model by performinglearning processing related to the ear model. The ear model learningunit 133 stores the generated ear model in the ear model storage unit122.

Exemplary learning processing executed by the reception unit 132 and theear model learning unit 133 will be described below with reference toFIG. 5. FIG. 5 is a diagram illustrating exemplary learning processingrelated to an ear model according to the present disclosure.

As illustrated in FIG. 5, the reception unit 132 receives data that iscollected from an ear form and scanned, and stores the received data inthe ear form information storage unit 121. In addition, the receptionunit 132 transfers the received data to the ear model learning unit 133(step S16).

The ear model learning unit 133 homogenizes the acquired ear form dataand generates homogenized data of the ear form (step S17). Thehomogenization is unification of the number of apexes and theconfiguration of polygons of a 3D model with those of a reference 3Dmodel. In this case, it is needed to pay attention not to have shapechange through the homogenization. In addition, the ear model learningunit 133 performs principal component analysis on the homogenized data(step S18). Accordingly, the ear model learning unit 133 generates amodel (ear model) that calculates, from the shape of an ear, an earparameter indicating the shape of the ear. The ear model learning unit133 stores the generated ear model in the ear model storage unit 122.

Description continues with reference to FIG. 3 again. The imagegeneration unit 134 generates an image including a content image of anear. For example, the image generation unit 134 randomly generates anear parameter and generates the 3D model of an ear by inputting thegenerated ear parameter to an ear model. In addition, the imagegeneration unit 134 randomly generates parameters (hereinafter referredto as “image generation parameters”) such as the texture (for example,skin color) of the generated 3D model, the rendering quality (such asimage quality) thereof, and the camera angle thereof at CG rendering.Then, the image generation unit 134 performs rendering with combinationof the generated 3D model and the plurality of image generationparameters as appropriate, thereby generating CG images among which theear shape, the skin color, and the like are different in variousmanners.

In estimation processing to be described later, an image transmittedfrom a user is used for the processing, and it is assumed that the skincolor of the user, the angle of an ear at image capturing, and the likeare highly likely to be different in various manners among imagestransmitted from the user. Thus, in the processing, it is needed toaccurately perform image recognition of a content image of an ear invarious images transmitted from the user, which is a problem. The imagegeneration unit 134 generates a large number of images corresponding tovarious situations as described above to improve the accuracy of imagerecognition, thereby solving the above-described problem.

The ear parameter learning unit 135 generates an ear parameterestimation model by learning the relation between an image including acontent image of an ear and an ear parameter. The ear parameter learningunit 135 corresponds to a first learning unit according to the presentdisclosure. An image including a content image of an ear may be an imageobtained by actually capturing an ear of a person or may be a CG imagegenerated based on an ear parameter as described later.

For example, the ear parameter learning unit 135 generates an earparameter estimation model by learning the relation between an earparameter and an ear image obtained by rendering three-dimensional dataobtained by synthesizing three-dimensional data of an ear generatedbased on the ear parameter with three-dimensional data of a head.Specifically, the ear parameter learning unit 135 learns the relationbetween a CG image generated by the image generation unit 134 and an earparameter. As described above, the image generation unit 134 generatesthe CG image based on a randomly or regularly set ear parameter, andthus the ear parameter is uniquely determined for the CG image.Accordingly, by learning the relation between an input CG image and anear parameter, the ear parameter learning unit 135 can generate, when animage is input, a model that outputs an ear parameter corresponding to acontent image of an ear included in the image. Note that the earparameter learning unit 135 does not necessarily use, for learning, anear image obtained by rendering three-dimensional data synthesized witha head. Specifically, the ear parameter learning unit 135 may generatean ear parameter estimation model by learning the relation between anear parameter and an ear image obtained by rendering onlythree-dimensional data of an ear generated based on the ear parameter.

In addition, the ear parameter learning unit 135 generates an earparameter estimation model by learning the relation between a pluralityof ear images among which the texture of three-dimensional data of anear or a head, the camera angle thereof in rendering, the luminancethereof in rendering, or the like is different and an ear parametercommon to the plurality of ear images. In this manner, the ear parameterlearning unit 135 performs learning by using ear images of variousaspects, and thus can generate a reliable and robust model that canaccurately output an ear parameter when any image is input (for example,can perform reliable estimation for any change of information other thanan ear parameter included in an input image).

Exemplary learning processing executed by the image generation unit 134and the ear parameter learning unit 135 will be described below withreference to FIG. 6. FIG. 6 is a diagram illustrating exemplary learningprocessing related to an ear parameter estimation model according to thepresent disclosure.

As illustrated in FIG. 6, the image generation unit 134 refers to theear model storage unit 122 (step S21) and acquires an ear model. Then,the image generation unit 134 generates a random number corresponding toan ear parameter and random numbers corresponding to CG texture, arendering camera angle, and the like (step S22). In other words, theimage generation unit 134 generates various parameters (image generationparameters) for generating an ear image.

Then, the image generation unit 134 acquires an ear parameter among theimage generation parameters (step S23) and reconstructs a 3D model of anear by inputting the acquired ear parameter to the ear model (step S24).

Subsequently, the image generation unit 134 acquires a parameter such asthe CG texture among the image generation parameters (step S25) andprovides the 3D model with 3DCG rendering by inputting the acquiredparameter (step S26). Note that a head used in the rendering is, forexample, an average head of a plurality of persons (for example, aplurality of persons for which ear forms are collected), the 3D model ofa head used as a sample, or the like. Similarly to an ear model, the 3Dmodel of a head may be generated by homogenizing 3D data obtained byperforming 3D scanning of the heads of a plurality of persons. In thiscase, the image generation unit 134 can randomly generate the head 3Dmodel by performing parameter generation with a random number. Note thatthe image generation unit 134 may generate, for texture, a texture modelgenerated by the same method, thereby generating various kinds oftexture with random numbers.

Accordingly, the image generation unit 134 generates an image (earimage) including a content image of an ear. Note that the imagegeneration unit 134 can generate a plurality of ear images from one 3Dmodel by changing a parameter such as texture in various manners.

An exemplary ear image generated by the image generation unit 134 willbe described below with reference to FIG. 7. FIG. 7 is a diagramillustrating exemplary ear image generation processing according to thepresent disclosure.

The image generation unit 134 generates a 3D model indicating thethree-dimensional shape of an ear by using a randomly generated earparameter (step S41). Then, the image generation unit 134 acquires ahead 3D model generated based on data indicating the three-dimensionalshape of an average head of a plurality of persons (step S42). Then, theimage generation unit 134 generates the 3D model of a pseudo person bysynthesizing the ear 3D model generated at step S41 and the head 3Dmodel acquired at step S42.

Subsequently, the image generation unit 134 performs processing(rendering) of generating a two-dimensional image from the generated 3Dmodel by performing pseudo image capturing of the 3D model. For example,the image generation unit 134 sets, as a pseudo image capturing angle,the angle of the front surface of the ear in the 3D model and generatesan image in which a content image of the ear is substantially at thecenter.

The image generation unit 134 changes the CG texture (specifically, skincolor or the like), the rendering quality (resolution or the like), theposition of the ear relative to the head, and the like in variousmanners by randomly inputting image generation parameters into the 3Dmodel. Accordingly, the image generation unit 134 can generate a largenumber of images among which the skin color and the like are different(step S43).

An image group 20 represents a plurality of ear images generated by theimage generation unit 134. In this manner, the image generation unit 134generates a larger number of various kinds of ear images, therebyimproving ear image recognition accuracy to be described later.

Description continues with reference to FIG. 6 again. The imagegeneration unit 134 stores the generated ear images in the ear imagestorage unit 123 (step S27). Note that the image generation unit 134stores, in the ear image storage unit 123 in association with eachgenerated ear image, the image generation parameter when the image isgenerated (step S28). Accordingly, the image generation unit 134 canhold, as learning data, a larger number of ear images each associatedwith an ear parameter. For example, the image generation unit 134 canhold, as learning data, a large number of ear images each associatedwith an ear parameter.

Subsequently, the ear parameter learning unit 135 refers to the earimage storage unit 123 (step S29) and acquires each ear image and thecorresponding ear parameter. Then, the ear parameter learning unit 135generates an ear parameter estimation model by learning the relationbetween the ear image and the ear parameter. The ear parameter learningunit 135 stores the generated ear parameter estimation model in the earparameter estimation model storage unit 124 (step S30).

The ear parameter estimation model is generated by using, for example, aconvolutional neural network that is useful for extracting thecharacteristic amount of an image. Note that a cost expression (costfunction) in learning is given by, for example, Expression (1) below.

L(α_(true),α_(est))=∥A _(ear)α_(true) −A _(ear)α_(est)∥₂ ²  (1)

In Expression (1), “α_(true)” represents the true value of an earparameter, “α_(est)” represents an estimated value of the ear parameter,and “A_(car)” represents an ear model obtained by principal componentanalysis. The distance function on the right hand side represents the L2norm (Euclidean distance). Note that the true value of an ear modelparameter may be, for example, a parameter representing an ear of aperson, which is measured at ear form collection. Specifically, the truevalue is an ear parameter used to generate an ear image, and theestimated value is a value output when the ear image is input to an earparameter estimation model in learning. As learning processing, theinformation processing device 100 updates a coefficient that minimizesthe value of the cost expression for a current estimated value.

The ear parameter estimation model generated by the ear parameterlearning unit 135 will be described below with reference to FIG. 8. FIG.8 is a diagram for describing the ear parameter estimation modelaccording to the present disclosure.

When having acquired an ear image 30, the information processing device100 inputs the acquired ear image 30 to the ear parameter estimationmodel. The ear parameter estimation model has, for example, thestructure of a convolutional neural network, divides the input ear image30 into rectangular parts each having several pixels, and calculates acharacteristic amount representing the ear image 30. Finally, the earparameter estimation model outputs, as the characteristic amountrepresenting the ear image 30, an ear parameter corresponding to thecontent image of the ear included in the ear image 30 (step S45).

Note that the information processing device 100 can reconstruct an earform (3D model) corresponding to the ear included in the ear image 30 byinputting the output ear parameter to an ear model. A 3D model 40illustrated in FIG. 8 is a 3D model through which the ear included inthe ear image 30 is reconstructed by CG based on an ear parameter.

Description continues with reference to FIG. 3 again. The HRTF learningunit 136 learns the relation between information related to the shape ofan ear and an HRTF, thereby generating various models related to theHRTF. For example, the HRTF learning unit 136 learns the relationbetween an image including a content image of an ear and an HRTFcorresponding to the ear, thereby generating a learned model thatcalculates the HRTF. The HRTF learning unit 136 corresponds to a secondlearning unit according to the present disclosure.

For example, the HRTF learning unit 136 performs acoustic simulation forthree-dimensional data obtained by synthesizing three-dimensional dataof an ear generated based on an ear parameter and three-dimensional dataof a head and learns the relation between an HRTF obtained through theacoustic simulation and the ear parameter, thereby generating a learnedmodel.

Alternatively, the HRTF learning unit 136 may compress the amount ofinformation of the HRTF obtained through the acoustic simulation andlearn the relation between the compressed HRTF and the ear parameter,thereby generating a learned model.

Alternatively, the HRTF learning unit 136 may set a hearing point ofthree-dimensional data of an ear generated based on the ear parameterand perform the acoustic simulation by using the set hearing point. Thehearing point is a virtually set position where it is assumed that aperson hears sound. For example, the position of the hearing pointcorresponds to a position (such as the entrance of an external ear canalof a dummy head) at which a microphone is installed in a dummy headmicrophone.

As for each above-described processing of the HRTF learning unit 136,FIG. 9 illustrates the process of processing of generating variousmodels related to an HRTF. FIG. 9 is a diagram illustrating an outlineof the process of processing of generating various models related to anHRTF.

FIG. 9 illustrates an example in which the HRTF learning unit 136performs predetermined learning processing based on an image transmittedfrom a user. In this case, the user performs image capturing of an ear(head including the ear, to be precise) of the user by using the userterminal 10 (step S51). Thereafter, the user terminal 10 performspreprocessing of specifying a range including a content image of the earin a captured picture and cutting out the specified range to acquire anear image (step S52).

Thereafter, the HRTF learning unit 136 calculates, by using an earparameter estimation model, the ear parameter of the ear included in theear image transmitted from the user (step S53). In addition, the HRTFlearning unit 136 reconstructs the 3D model of the ear based on the earparameter and combines a head 3D model to the reconstructed ear, therebygenerating the 3D model of the person (step S54).

Subsequently, the HRTF learning unit 136 performs acoustic simulationfor the generated 3D model to calculate the personalized HRTF of the 3Dmodel (step S55). Accordingly, the HRTF learning unit 136 can obtainlearning data in which the ear included in the ear image transmittedfrom the user is associated with the personalized HRTF.

Note that, in the example illustrated in FIG. 9, learning data in whicha personalized HRTF obtained through acoustic simulation is associatedwith data of an ear is generated, but the HRTF learning unit 136 doesnot necessarily need to obtain a personalized HRTF through acousticsimulation in some cases. For example, when the personalized HRTF (HRTFobtained by using a measurement device in an anechoic room or the like)of a person for which an ear form is collected is obtained, the HRTFlearning unit 136 may acquire learning data in which the measuredpersonalized HRTF is associated with the ear form (ear parameter) of theperson.

At acoustic simulation, the HRTF learning unit 136 automatically sets ahearing point in the 3D model of a person. This will be described belowwith reference to FIG. 10. FIG. 10 is a diagram for describing 3D modelreconstruction according to the present disclosure.

The HRTF learning unit 136 reconstructs an ear 3D model based on arandomly generated ear parameter (step S71). Subsequently, the HRTFlearning unit 136 generates the 3D model of a person by combining theear 3D model with a head 3D model (step S72). In addition, the HRTFlearning unit 136 sets a hearing point 60 of a sound source based on theshape of an ear in the 3D model, (step S73). For example, the HRTFlearning unit 136 may automatically set a hearing point by learning, inadvance, information in which the shape of an ear and the position of ahearing point of a sound source are associated with each other. Forexample, at a timing when the 3D model is generated, the HRTF learningunit 136 estimates the position of a hearing point in the 3D model,thereby automatically setting the hearing point. A hearing pointcorresponds to, for example, the external ear canal of an ear, andtypically, the position thereof can be estimated based on the shape ofthe ear.

Thereafter, the HRTF learning unit 136 re-meshes the generated 3D modelto satisfy a constraint on 3D model calculation in acoustic simulation(step S74). This is because, in 3D model simulation, the upper limit ofthe total number of polygons, the length of an edge connecting apexes,and the like are determined due to simulation conditions. In otherwords, before simulation with the generated 3D model, the HRTF learningunit 136 processes the 3D model by re-meshing as appropriate to satisfya calculation constraint so that the simulation can be appropriatelyperformed. Then, the HRTF learning unit 136 performs acoustic simulationfor the generated 3D model and the set hearing point 60, therebycalculating a personalized HRTF (step S75).

Subsequently, a detailed process of processing of generating modelsrelated to an HRTF will be described below with reference to FIG. 11.FIG. 11 is a diagram for describing details of the processing ofgenerating models related to an HRTF.

As illustrated in FIG. 10, the HRTF learning unit 136 performs headsynthesis (step S81) and then performs acoustic simulation (step S82).The HRTF learning unit 136 analyzes measurement data obtained throughthe acoustic simulation (HRTF postprocessing) and calculates a numericalvalue representing a personalized HRTF (step S83). Note that the HRTFpostprocessing is, for example, calculation of an HRTF through Fouriertransform of a head-related impulse response (HRIF) obtained through theacoustic simulation.

Then, the HRTF learning unit 136 refers to the HRTF processing modelstorage unit 125 (step S84) and inputs the calculated HRTF to a model(HRTF processing model) for processing the HRTF. Accordingly, the HRTFlearning unit 136 obtains an HRTF having a reduced dimension (step S85).Specifically, the HRTF learning unit 136 outputs, from the HRTFprocessing model, an HRTF parameter as the HRTF having a reduceddimension (step S86).

In this manner, the HRTF learning unit 136 performs processing by usingthe HRTF parameter having a reduced dimension instead of directly using,in the processing, the HRTF obtained through the acoustic simulation.This is because the HRTF is a function having an extremely large numberof dimensions and thus leads to a large computation processing load whendirectly used in model generation processing and calculation processing.

The HRTF learning unit 136 stores data (data on which the head synthesisis based, such as an ear parameter) related to a head for which theacoustic simulation is performed, and the calculated HRTF parameter inthe HRTF learning data storage unit 126 in association with each other(step S87).

Thereafter, the HRTF learning unit 136 newly generates a differentrandom ear parameter (step S88) and performs head synthesis by using thenewly generated ear parameter (step S89). The HRTF learning unit 136repeats steps S81 to S89 to collect learning data necessary forlearning.

Thereafter, when a sufficient amount of learning data is accumulated,the HRTF learning unit 136 refers to the HRTF learning data storage unit126 (step S90) and learns the relation between an ear parameter and anHRTF (HRTF parameter, to be precise) (step S91). Through the learning,the HRTF learning unit 136 generates an HRTF learning model forobtaining an HRTF directly from an ear parameter, and stores thegenerated HRTF learning model in the HRTF learning model storage unit127.

Subsequently, the relation between an HRTF and an HRTF parameter will bedescribed below with reference to FIG. 12. FIG. 12 is a diagram fordescribing HRTF compression and decompression according to the presentdisclosure.

As illustrated in FIG. 12, the HRTF learning unit 136 performs fastFourier transform (FFT) of an HRIFHRTF ((for example, 1000directions×500 taps in the example of FIG. 12) obtained through acousticsimulation (step S101). Through this processing, the HRTF learning unit136 extracts an amplitude characteristic (step S102) and performs, forexample, thinning processing of a frequency component for which hearingsensitivity is low or the like (step S103). Specifically, the HRTF canbe expressed as a function HRTF (θ, ϕ, f) of angles (denoted by θ and ϕ)and a frequency (denoted by f). In this case, when k represents thenumber of bins of the frequency, the frequency f input to the functionis expressed as f_(k)=f₀, f₁, f₂, . . . , f_(k-1). In other words, theHRTF has complex k dimensions for one direction and one ear. The HRTF atthe Nyquist frequency (f_(k/2)) or higher is obtained by folding thecomplex conjugate at the frequency f_(k/2) or lower, and thus in theinformation processing, only (k/2)+1 frequencies from f0=0 to theNyquist frequency (f_(k/2)) can be used as a frequency bin. Absolutevalues can be used for at least one frequency bin. For example, when allfrequencies of f₀ to f_(k/2) are converted into absolute values, afunction H₂ after the conversion is given by Expression (2) below.

H ₂(θ,ϕ,k)=|HRTF(θ,ϕ,f _(k))|(f _(k) =f ₀ ,f ₁ , . . . ,f _(k/2))  (2)

Specifically, the HRTF learning unit 136 can compress the dimension ofthe original HRTF to the dimension of a real number (k/2)+1. The HRTFlearning unit 136 can achieve further reduction to a dimension smallerthan (k/2)+1 by performing frequency compression for H₂ in theabove-described expression (2). Various kinds of known methods exist forthe dimension compression. For example, the HRTF learning unit 136 usesa method of performing cepstral conversion on the function to acquireonly frequency bins in a number equal to or larger than one and smallerthan (k/2)+1. In an example, the HRTF learning unit 136 calculates anaverage value of a plurality of frequency bins and performs dimensionreduction based on the average value. For example, when a frequency binis given by Expression (3) below (a_(l), L, and l are integers equal toor larger than zero), a new function H₃ is given by Expression (4) belowby using 1 that satisfies f_(a1)≤f′₁<f_(al+1).

$\begin{matrix}{{0 \leq a_{0} < a_{1} < {a_{2}\mspace{14mu}\ldots}\mspace{14mu} < a_{L - 1} \leq {K/2}},{L < {{K/2} + 1}}} & (3) \\{{H_{3}( {\theta,\phi,1} )} = {\frac{1}{a_{l + 1} - a}{\sum\limits_{k = a_{l}}^{a_{l + 1} - 1}{H_{2}(k)}}}} & (4)\end{matrix}$

Accordingly, the HRTF learning unit 136 can reduce the function H₂expressed in (K/2)+1 dimensions to L dimensions. Note that the method ofcalculating the average value is not limited to that described above,but the average value may be calculated by using, for example,root-mean-square or weighted average. Accordingly, the HRTF is reducedto, for example, 1000 directions×50 dimensions. Note that when thedimension reduced by the function H₃ is restored (for example, step S110to be described later), the HRTF learning unit 136 can restore thedimension by using various kinds of methods such as linear interpolationor spline interpolation. It is expected that a function H′₂ having therestored dimension has a characteristic smoother than that of thefunction H₂, and H′₂(θ, ϕ, k) that provides smaller hearing influencecan be obtained by contriving the way of selecting a_(l). For example,a_(l) may be selected so that the frequency interval between f_(al) andf_(al+1) is larger at a higher frequency.

The HRTF learning unit 136 further performs processing of sphericalharmonic fitting on the HRTF having a reduced dimension, therebycompressing the amount of information to 50 coefficient×50 dimensions(step S104). The spherical harmonic fitting is spatial directionalfitting for each compressed frequency by using spherical harmonics. Therelation between the HRTF and the spherical harmonics is given byExpression (5) below.

$\begin{matrix}{{H( {\theta,\phi,f} )} = {\sum\limits_{n = 0}^{\infty}{\sum\limits_{m = {- n}}^{n}{{h_{nm}(f)}{Y_{n}^{m}( {\theta,\phi} )}}}}} & (5)\end{matrix}$

As in the above-described expression (5), a spherical harmonics Y isexpressed by a coefficient h_(nm)(f). A dimension number n in this casecan be truncated at a finite value N to set the dimension of thecoefficient h_(nm)(f) to be smaller than the dimension number (directionnumber) of the original HRTF. This means discarding of spatially toofine amplitudes that are unnecessary for perception of a person andacquisition of only a smooth shape. Note that, for example, aleast-square method is used to calculate a vector h=(h₀₀, h₁₋₁, . . .)^(T) of the coefficient h_(nm).

E=∥H−Yh∥ ² +λ∥h∥ ²  (6)

Specifically, in the above-described expression (6), h that minimizes Eon the left-hand side is calculated when Y represents a sphericalharmonic matrix and H represents a spherical harmonic matrix. Note thatthe second term on the right hand side of the above-described expression(6) is a regularization term, and thus an arbitrary value may beselected as λ (for example, λ may be zero). Accordingly, theabove-described h is given by Expression (7) below.

h=(Y ^(T) Y+λI)⁻¹ Y ^(T) H  (7)

By using the above-described expression (7), the HRTF learning unit 136can obtain each h corresponding to a necessary frequency. In addition,the HRTF learning unit 136 performs dimension compression by principalcomponent analysis so that the amount of information of the HRTF can beexpressed in several hundred dimensions approximately (step S105). Thisinformation is an HRTF parameter (step S106).

Note that when the spherical harmonic fitting is performed after thefrequency thinning, the value of the above-described f is arepresentative frequency after the thinning. The HRTF learning unit 136may perform the frequency thinning after the spherical harmonic fitting.The method of spatially compressing dimension is not limited to linearcombination such as the spherical harmonics and principal componentanalysis but may be any method. For example, the HRTF learning unit 136may use a non-linear method such as kernel principal component analysis.The HRTF learning unit 136 may change the truncating order N of thespherical harmonics in accordance with the frequency f and use the valueof N(f). There may be a coefficient h_(nm) that is not used in thedimension number of 0 to N nor the order. The HRTF learning unit 136 maycalculate right and left HRTFs or may calculate the right and left HRTFsafter conversion into the sum or difference of the right and left HRTFs.A fitting target HRTF may be provided with various kinds of conversioninto, for example, the absolute value of amplitude and the logarithmrepresentation thereof.

Subsequently, the HRTF learning unit 136 can decode the HRTF byperforming processing in a process opposite to that of steps S101 toS106. First, the HRTF learning unit 136 acquires the HRTF parameter(step S107) and restores dimension compression by principal componentanalysis (step S108). In addition, the HRTF learning unit 136 performsspherical harmonic reconstruction processing (step S109) and performsfrequency interpolation (step S110). In addition, the HRTF learning unit136 obtains an amplitude characteristic (step S111) and performs minimumphase restoration (step S112). The minimum phase restoration may employvarious kinds of known methods. For example, the HRTF learning unit 136performs inverse fast Fourier transform (IFFT) of the logarithm of afunction H′1(θ, ϕ, k) restored by folding the above-described functionH′2 at the Nyquist frequency or higher, and determines the real part ofa result of the conversion. In addition, appropriate wind processing isperformed in this region, inverse fast Fourier transform of theexponential function thereof is performed, and the real part isdetermined, thereby performing the minimum phase restoration. Forexample, each relational Expression (8) as described below holds.

$\begin{matrix}{\mspace{79mu}{{H_{1}^{\prime}( {\theta,\phi,k} )} = \{ {{\begin{matrix}{{H_{2}^{\prime}( {\theta,\phi,k} )}\ } & ( {{{for}\mspace{14mu} 0} \leq k \leq {K/2}} ) \\{{H_{2}^{\prime}( {\theta,\phi,\ {K - k}} )}\ } & ( {{{for}\mspace{14mu}{K/2}} < k \leq {K - 1}} )\end{matrix}\mspace{20mu}{H_{c}( {\theta,\phi} )}} = {{{Re}\{ {F^{- 1}\lbrack {H_{1}^{\prime}( {\theta,\phi} )} \rbrack} \}{H_{w}( {\theta,\phi,k} )}} = \{ {{\begin{matrix}{{H_{c}( {\theta,\phi,k} )}\ } & ( {{{{for}\mspace{14mu} K} = 0},{K/2}} ) \\{{2{H_{c}( {\theta,\phi,k} )}}\ } & ( {{{for}\mspace{14mu} 1} \leq k \leq {{K/2} - 1}} ) \\{0\ } & ( {{{for}\mspace{14mu}{K/2}} \leq k \leq {K - 1}} )\end{matrix}\mspace{20mu}{H_{m}( {\theta,\phi} )}} = {{Re}\{ {F^{- 1}\lbrack {\exp( {H_{w}( {\theta,\phi} )} )} \rbrack} \}}} }} }} & (8)\end{matrix}$

Note that the HRTF learning unit 136 may add an estimated interauraltime difference (ITD) or an ITD prepared in advance to the right HRIR(h_(m)) and the left HRIR (h_(m)) subjected to the minimum phaserestoration. Note that an ITD is calculated by, for example, Expressions(9) and (10) below based on the group delay difference between the rightHRIR and the left HRIR.

$\begin{matrix}{{{ITD}_{g}( {\theta,\phi,f} )} = {\frac{1}{2\pi}( {\frac{d\;\varphi_{L}}{df} - \frac{d\;\varphi_{R}}{df}} )}} & (9) \\{\varphi_{L} = {\arctan( \frac{{Im}( {{HRTF}_{L}(f)} )}{{Re}( {{HRTF}_{L}(f)} )} )}} & (10)\end{matrix}$

Alternatively, the ITD may be calculated by determining a time-axialmutual correlation between the right and left and defining the ITD to bea time at which a correlation coefficient is maximum. In this case, theITD is calculated by, for example, Expressions (11) and (12) below.

$\begin{matrix}{(\tau) = {a\frac{\int_{- \infty}^{+ \infty}{{h_{L}( {t + \tau} )}{h_{R}(t)}{dt}}}{\sqrt{\int_{- \infty}^{+ \infty}{{h_{L}^{2}(t)}{{dt} \cdot {\int_{- \infty}^{+ \infty}{{h_{R}^{2}(t)}{dt}}}}}}}}} & (11) \\{{{ITD}_{corr}( {\theta,\phi} )} = \tau_{\max}} & (12)\end{matrix}$

For example, the HRTF learning unit 136 uses a relational expressionsuch as Expression (13) below to retard the left HRIR relative to theright HRIR by d samples.

$\begin{matrix}{{h_{L}(i)} = \begin{Bmatrix}0 & {{{for}\mspace{14mu} i} < d} \\{h_{m,L}( {i - d} )} & {{{for}\mspace{14mu} i} \geq d}\end{Bmatrix}} & (13)\end{matrix}$

In this case, h_(L) in the above-described expression (13) is an impulseresponse longer than h_(m,L) by d, and a length equal to that of h_(m,L)is achieved by deleting the second half of the above-describedexpression (13). In this case, the HRTF learning unit 136 may perform,for example, optional window, rectangular window, or Hanning windowprocessing. Note that the HRTF learning unit 136 may not only add theITD for each direction but also add a delay including a relative timedifference between directions in the entire space. In this case, theHRTF learning unit 136 acquires not only the ITD but also informationindicating the relative time difference between directions. When the ITDis a function of frequency, the HRTF learning unit 136 may add the ITDin the frequency domain or may calculate a representative value or anaverage value and then add the ITD. The HRTF learning unit 136 acquireseach HRIR in the original format and then performs inverse fast Fouriertransform to obtain the HRTF.

In this manner, the HRTF learning unit 136 may perform compression tothe HRTF parameter having an information amount smaller than that of theoriginal HRTF and perform, in the compressed format, HRTF learning modelgeneration processing and HRTF calculation processing to be describedlater. In addition, as described above, the HRTF compression performsdimension reduction utilizing a hearing sense characteristic by, forexample, using less sensitivity of human hearing to phase change orperforming preferential thinning of a frequency that is unlikely toaffect hearing. Accordingly, the HRTF learning unit 136 can increaseinformation processing speed without losing hearing stationary as anHRTF characteristic.

Description continues with reference to FIG. 3 again. The estimationunit 140 performs processing of estimating an HRTF corresponding to auser based on an image transmitted from the user.

The acquisition unit 141 acquires an image including a content image ofan ear of the user. For example, the acquisition unit 141 acquires acutout ear image around an ear of the user from an image captured by theuser terminal 10.

The acquisition unit 141 may input the acquired ear image to an earparameter estimation model to acquire an ear parameter indicating acharacteristic of the ear included in the image.

The calculation unit 142 calculates, based on the image acquired by theacquisition unit 141, an HRTF (personalized HRTF) corresponding to theuser by using a learned model (HRTF learning model) having learned tooutput an HRTF corresponding to an ear when an image including a contentimage of the ear is input.

Specifically, the calculation unit 142 calculates the personalized HRTFcorresponding to the user by inputting the ear parameter acquired by theacquisition unit 141 into the HRTF learning model.

Note that when calculating the personalized HRTF, the calculation unit142 may first calculate an HRTF parameter and then calculate the HRTF bydecoding the calculated HRTF parameter. In this manner, the calculationunit 142 can increase processing speed by performing a series of piecesof processing while the amount of information of the HRTF is compressed.In addition, the calculation unit 142 can avoid outputting of an oddHRTF that is not expressed in an HRTF reduction model, and thus canperform reliable outputting.

The provision unit 143 provides the HRTF calculated by the calculationunit 142 to the user through the network N.

The process of processing of estimating an HRTF from an image will bedescribed below with reference to FIG. 13. FIG. 13 is a diagramillustrating the process of HRTF estimation processing according to thepresent disclosure.

FIG. 13 illustrates an example in which the estimation unit 140performs, based on an image transmitted from a user, the processing ofestimating an HRTF corresponding to an ear included in the image. Inthis case, the user performs image capturing of an ear (head includingthe ear, to be precise) of the user by using the user terminal 10 (stepS131). Thereafter, the user terminal 10 performs preprocessing ofspecifying a range including a content image of the ear in a capturedpicture and cutting out the specified range to acquire an ear image(step S132).

When having acquired the ear image transmitted from the user, theacquisition unit 141 inputs the acquired ear image to a learned model.Specifically, the acquisition unit 141 inputs the ear image to an earparameter estimation model. The ear parameter estimation model outputs,as a characteristic amount indicating the ear image, an ear parametercorresponding to the content image of the ear included in the ear image.Accordingly, the acquisition unit 141 acquires the ear parametercorresponding to the image (step S133).

The calculation unit 142 inputs the acquired ear parameter into an HRTFlearning model and calculates a personalized HRTF corresponding to theear image (step S133). The provision unit 143 provides (transmits) thecalculated personalized HRTF to the user terminal 10 from which theimage is transmitted (step S134).

In this manner, when various kinds of models are generated by thelearning unit 131, the information processing device 100 can perform theear image acquisition to the personalized HRTF provision as a series ofpieces of processing. Accordingly, the information processing device 100can improve convenience related to the HRTF provision for the user.

Note that although a combination of an ear parameter estimation modeland an HRTF learning model is illustrated as an exemplary learned modelin the example of FIG. 13, a learned model combination is not limited tothis example. The learned model may be an individual combination of theear parameter estimation model and the HRTF learning model or may beconfigured as one model that performs processing corresponding to theear parameter estimation model and the HRTF learning model.

[1-4. Configuration of User Terminal According to First Embodiment]

As illustrated in FIG. 13, image capturing of a side face of a user andgeneration of an ear image are performed by the user terminal 10 in thefirst embodiment. The configuration of the user terminal 10 according tothe first embodiment will be described below. FIG. 14 is a diagramillustrating an exemplary configuration of the user terminal 10according to the first embodiment of the present disclosure. Asillustrated in FIG. 14, the user terminal 10 includes a communicationunit 11, an input unit 12, a display unit 13, a sensing unit 14, astorage unit 15, and a control unit 16.

The communication unit 11 is achieved by, for example, a NIC. Thecommunication unit 11 is connected with the network N in a wired orwireless manner and transmits and receives information to and from theinformation processing device 100 and the like through the network N.

The input unit 12 is an input device configured to receive variousoperations from a user. For example, the input unit 12 is achieved by anoperation key or the like included in the user terminal 10. The displayunit 13 is a display device for displaying various kinds of information.For example, the display unit 13 is achieved by a liquid crystaldisplay. Note that when the user terminal 10 employs a touch panel, partof the input unit 12 and the display unit 13 are integrated with eachother.

The sensing unit 14 collectively refers to various sensors and sensesvarious kinds of information related to the user terminal 10.Specifically, the sensing unit 14 senses an operation on the userterminal 10 by the user, position information of the user terminal 10,information related to an instrument connected with the user terminal10, environment at the user terminal 10, and the like.

The sensing unit 14 includes, as an exemplary sensor, a lens and animage sensor for performing image capturing. Specifically, for example,the sensing unit 14 functions as a camera when an application configuredto operate an image capturing function is activated by the user.

The storage unit 15 stores various kinds of information. The storageunit 15 is achieved by a storage device, for example, a semiconductormemory element such as a RAM or a flash memory, a hard disk, or anoptical disk. The storage unit 15 stores, for example, an image capturedby the user.

The control unit 16 is a controller achieved by, for example, a CPU oran MPU executing, by using the RAM as a work area, various computerprograms stored in a storage device inside the user terminal 10.Alternatively, the control unit 16 is a controller achieved by anintegrated circuit such as an ASIC or an FPGA.

As illustrated in FIG. 14, the control unit 16 includes an acquisitionunit 161, a preprocessing unit 162, a transmission unit 164, and areception unit 165 and achieves or executes functions and effects ofinformation processing described below. The preprocessing unit 162includes a posture detection unit 163A and an ear detection unit 163B.Note that the internal configuration of the control unit 16 is notlimited to the configuration illustrated in FIG. 14 but may be anyconfiguration with which the information processing to be describedlater is performed.

The acquisition unit 161 acquires various kinds of information. Forexample, the acquisition unit 161 acquires an image captured by thesensing unit 14.

The posture detection unit 163A reads the image acquired by theacquisition unit 161 and detects the posture of the user included in theimage.

The ear detection unit 163B detects a range (ear content image)including an ear of the user included in the image based on the postureof the user detected by the posture detection unit 163A. Specifically,the ear detection unit 163B specifies a content image of an ear of theuser in an image including a content image of the entire head of theuser and detects the specified range as an ear image.

For example, the ear detection unit 163B specifies a range including acontent image of an ear based on the relation between each feature pointof the head of the user included in the entire image and the posture ofthe user.

When the range including a content image of an ear cannot be specifiedbased on the relation between each feature point of the head of the userincluded in the entire image and the posture of the user, the posturedetection unit 163A or the ear detection unit 163B may newly request theuser to acquire an image that is different from the entire image andincluding a content image of the entire head of the user. Specifically,the posture detection unit 163A or the ear detection unit 163B promptsthe user to perform image capturing again by displaying, on the displayunit 13, a message indicating that the information processing accordingto the present disclosure potentially cannot be appropriately performwith the image of a side face captured by the user. Note that theposture detection unit 163A or the ear detection unit 163B may promptthe user to perform image capturing again not only in a case in whichthe range including a content image of an ear cannot be specified butalso in a case in which, for example, a case in which a camera angleused in learning of an ear parameter estimation model and the posture ofthe user exceed certain threshold values. In addition, as thepreprocessing, the posture detection unit 163A or the ear detection unit163B may generate correction information that corrects the posture andposition of the user in an image instead of detecting the ear image ofthe user. The correction information is, for example, information thatinstructs the amount of rotating the range including a content image ofthe ear or the like in accordance with the tilt and rotation of afeature point of the head of the user. Such information is generatedbased on the posture of the user, the positional relation between a sideface of the user and a detected ear, and the like as described later. Inthis case, the posture detection unit 163A or the ear detection unit163B may specify a content image of an ear of the user by correctingrotation of the entire image based on the correction information and maydetect the specified range as an ear image. The posture detection unit163A or the ear detection unit 163B may transmit the entire imagetogether with the generated correction information to the informationprocessing device 100. In this case, the information processing device100 performs, at the own device, preprocessing of specifying a contentimage of an ear of the user by correcting rotation of the entire imagebased on the correction information transmitted together with the entireimage and detecting the specified range as an ear image.

The process of the preprocessing executed by the preprocessing unit 162(the posture detection unit 163A and the ear detection unit 163B) willbe described below with reference to FIG. 15. FIG. 15 is a diagramillustrating the process of detection processing according to thepresent disclosure.

As illustrated in FIG. 15, when a side face of the user is captured bythe user, the acquisition unit 161 acquires an entire image 50 (stepS141).

The posture detection unit 163A detects a side face of the user in theentire acquired image 50 (step S142). For example, the posture detectionunit 163A specifies a range including a content image of a side face ofthe user in the entire image 50 by using a known technology such ashuman face detection processing.

Then, the posture detection unit 163A detects a feature point includedin the side face of the user as illustrated with an image 51. Forexample, the posture detection unit 163A detects, in the side face,feature points such as a place (specifically, the apex of the nose ofthe user) protruding in the horizontal direction, the apex of the head,the position of the mouth, and the position of the jaw. The posturedetection unit 163A also detects, for example, the positions of an earand a sideburn of the user based on information of the boundary of hairand skin and the like. The posture detection unit 163A also detects theposition of an eye of the user or the like based on color information ofthe content image of the side face.

Then, the posture detection unit 163A detects the posture of the userbased on the detected feature points (step S143). For example, theposture detection unit 163A detects the posture of the head of the userbased on three-dimensional disposition of the feature points asillustrated with an image 54.

Such posture detection processing is processing to prevent the posturein an ear image transmitted by the user from largely deviating from theposture of a 3D model used at learning. This is because when an imageincluding a posture largely different from that of the 3D model istransmitted from the user terminal 10, the information processing device100 potentially cannot appropriately perform ear image recognition dueto deviation between the learning data and the transmitted ear image.

Thus, the posture detection unit 163A determines whether the differencebetween the average value of the angle of a head 3D model 55 used inlearning through rendering and an angle obtained from the image 54 isequal to or smaller than a predetermined threshold value, therebyperforming success determination of whether the user has appropriatelyperformed image capturing (step S144). For example, it is assumed that,in learning of an ear parameter estimation model, an angle ϕ between theorientation of a camera at rendering of the head 3D model 55 and a linesegment connecting the head apex and a predetermined position of an ear(for example, the entrance of the external ear canal) is equal to orsmaller than a predetermined numerical value. Similarly, it is assumedthat, in learning of an ear parameter estimation model, an angle θbetween the orientation of the camera and a line segment connecting theapex of the nose and a predetermined position of the ear is equal to orsmaller than a predetermined numerical value. This is because an earimage used in learning does not largely deviate from an imageillustrating a side face of a human to increase image recognitionaccuracy. Specifically, the posture detection unit 163A determineswhether, similarly to an image in learning, an image transmitted fromthe user has an angle with which the image is recognizable as an imageillustrating a side face of a human.

When having determined that the user has not appropriately performedimage capturing (for example, when the nose points downward beyond apredetermined threshold value at the face of the user), the posturedetection unit 163A performs processing of, for example, displaying amessage that requests execution of image capturing again and thenacquires a newly captured image (step S145).

When having determined that the user has appropriately performed imagecapturing (step S146), the ear detection unit 163B specifies a range 57including a content image of an ear in an image 56 and cuts out therange 57 (step S147). Accordingly, the ear detection unit 163B acquiresan ear image 58.

By performing the detection processing illustrated in FIG. 15, theinformation processing device 100 can determine whether an ear of theuser is tilted due to a poor image capturing state or the angle of theear is actually tilted, and then perform HRTF calculation.

In addition, the user terminal 10 cuts out an ear image from the entireimage of a side face as described above, and thus, not the entire image,in which the face of the user is included, but only the ear image can betransmitted to perform processing. Accordingly, the user terminal 10 canprevent leakage of private information, thereby increasing the securityof information processing. Note that the user terminal 10 is not limitedto the above-described detection method but may perform processing ofcutting out an ear image from the entire image of a side face bydetecting an ear of a user included in the image by using an imagerecognition technology by machine learning or the like.

Description continues with reference to FIG. 14 again. The transmissionunit 164 transmits the ear image generated based on the range detectedby the ear detection unit 163B to the information processing device 100.

The reception unit 165 receives a personalized HRTF provided by theinformation processing device 100. For example, the reception unit 165incorporates the received personalized HRTF into music and voice at asound playback application or the like, thereby achieving 3D acousticand the like optimized for an individual user.

2. Second Embodiment

Subsequently, the second embodiment will be described below. The firstembodiment describes above an example in which the user terminal 10generates an ear image by cutting out only a content image of an earfrom an image captured by a user. An information processing device 100Aaccording to the second embodiment performs, by the own device in placeof the user terminal 10, processing of cutting out only a content imageof an ear.

The configuration of the information processing device 100A according tothe second embodiment will be described below with reference to FIG. 16.FIG. 16 is a diagram illustrating an exemplary configuration of theinformation processing device 100A according to the second embodiment ofthe present disclosure. As illustrated in FIG. 16, the informationprocessing device 100A further includes a preprocessing unit 144 (aposture detection unit 145A and an ear detection unit 145B) in additionto the configuration of the first embodiment.

The posture detection unit 145A performs processing same as thatperformed by the posture detection unit 163A according to the firstembodiment. The ear detection unit 145B performs processing same as thatperformed by the ear detection unit 163B according to the firstembodiment. Accordingly, the information processing device 100Aaccording to the second embodiment executes, by the own device,preprocessing executed by the user terminal 10 according to the firstembodiment.

In the second embodiment, the acquisition unit 141 acquires the entireimage of a side face, which is captured by a user, from the userterminal 10. Then, the posture detection unit 145A and the ear detectionunit 145B generate an ear image based on the entire image by performingprocessing same as the processing described with reference to FIG. 15.The calculation unit 142 calculates a personalized HRTF based on the earimage generated by the posture detection unit 145A and the ear detectionunit 145B.

In this manner, with the information processing device 100A according tothe second embodiment, the user can receive personalized HRTF provisionby only capturing and transmitting an image. In addition, with theconfiguration of the second embodiment, since the preprocessing does notneed to be executed at the user terminal 10, it is possible to reduce,for example, a processing load on the user terminal 10. Moreover, sinceit can be assumed that, typically, processing speed is higher at theserver device (information processing device 100) than the user terminal10, it is possible to improve the entire speed of the informationprocessing according to the present disclosure with the configuration ofthe second embodiment. Note that when correction information istransmitted together with the entire image, the posture detection unit145A and the ear detection unit 145B may specify a content image of anear of the user by correcting rotation of the entire image based on thecorrection information included in the entire image and may detect thespecified range as an ear image.

3. Other Embodiments

Processing according to the above-described embodiments may be performedin various kinds of different forms other than the above-describedembodiments.

Among the pieces of processing described above in the embodiments, allor some pieces of processing described as automatically performedprocessing may be manually performed, or all or some pieces ofprocessing described as manually performed processing may beautomatically performed by a well-known method. In addition, informationincluding processing procedures, specific names, and various kinds ofdata and parameters, which are indicated in the above-described documentand drawings may be optionally changed unless otherwise stated. Forexample, various kinds of information illustrated in each drawing arenot limited to the illustrated information.

Components of devices illustrated in the drawings represent conceptualfunctions and are not necessarily physically configured as illustratedin the drawings. In other words, specific forms of dispersion andintegration of the devices are not limited to the illustrated forms, andall or some of the devices may be functionally or physically dispersedand integrated in optional units in accordance with various burdens anduse situations.

The above-described embodiments and modifications may be combined asappropriate without inconsistency of processing contents.

Effects described in the present specification are merely exemplary andnot restrictive, but any other effect may be achieved.

4. Hardware Configuration

An information instrument such as the information processing device 100or the user terminal 10 according to each above-described embodiment isachieved by, for example, a computer 1000 having a configuration asillustrated in FIG. 17. The following describes an example with theinformation processing device 100 according to the first embodiment.FIG. 17 is a hardware configuration diagram illustrating an exemplarycomputer 1000 configured to achieve functions of the informationprocessing device 100. The computer 1000 includes a CPU 1100, a RAM1200, a read only memory (ROM) 1300, a hard disk drive (HDD) 1400, acommunication interface 1500, and an input-output interface 1600. Thecomponents of the computer 1000 are connected with one another through abus 1050.

The CPU 1100 operates based on a computer program stored in the ROM 1300or the HDD 1400 and performs control of each component. For example, theCPU 1100 executes processing corresponding to various computer programsby burdening the computer programs stored in the ROM 1300 or the HDD1400 onto the RAM 1200.

The ROM 1300 stores a boot program such as a basic input/output system(BIOS) executed by the CPU 1100 at activation of the computer 1000, acomputer program dependent on hardware of the computer 1000, and thelike.

The HDD 1400 is a computer-readable recording medium in which a computerprogram executed by the CPU 1100, data used by the computer program, andthe like are recorded in a non-temporary manner. Specifically, the HDD1400 is a recording medium that records the information processingprogram according to the present disclosure, which is exemplary programdata 1450.

The communication interface 1500 is an interface for connecting thecomputer 1000 with an external network 1550 (for example, the Internet).For example, through the communication interface 1500, the CPU 1100receives data from another instrument and transmits data generated bythe CPU 1100 to another instrument.

The input-output interface 1600 is an interface for connecting thecomputer 1000 with an input-output device 1650. For example, the CPU1100 receives data from an input device such as a keyboard or a mousethrough the input-output interface 1600. The CPU 1100 transmits data toan output device such as a display, a speaker, or a printer through theinput-output interface 1600. The input-output interface 1600 mayfunction as a medium interface for reading a computer program or thelike recorded in a predetermined recording medium. The medium is, forexample, an optical recording medium such as a digital versatile disc(DVD) or a phase change rewritable disk (PD), a magneto-opticalrecording medium such as a magneto-optical disk (MO), a tape medium, amagnetic recording medium, or a semiconductor memory.

For example, when the computer 1000 functions as the informationprocessing device 100 according to the first embodiment, the CPU 1100 ofthe computer 1000 achieves the function of the control unit 130 or thelike by executing an information processing program loaded onto the RAM1200. The HDD 1400 stores the information processing program accordingto the present disclosure and data in the storage unit 120. Note thatthe CPU 1100 reads the program data 1450 from the HDD 1400 and executesthe program data 1450, but for example, may acquire these computerprograms from another device through the external network 1550.

Note that the present technology may be configured as described below.

(1)

An information processing device comprising:

an acquisition unit configured to acquire a first image including acontent image of an ear of a user; and

a calculation unit configured to calculate, based on the first imageacquired by the acquisition unit, a head-related transfer functioncorresponding to the user by using a learned model having learned tooutput a head-related transfer function corresponding to an ear when animage including a content image of the ear is input.

(2)

The information processing device according to (1), wherein

the acquisition unit acquires an ear parameter that is a variablerepresenting a characteristic of the ear included in the first image,and

the calculation unit calculates the head-related transfer functioncorresponding to the user by inputting the ear parameter to the learnedmodel.

(3)

The information processing device according to (2), wherein theacquisition unit acquires the ear parameter of the ear included in thefirst image by using an ear parameter estimation model having learned tooutput an ear parameter corresponding to an ear when an image includinga content image of the ear is input.

(4)

The information processing device according to (3), further comprising afirst learning unit configured to generate the ear parameter estimationmodel by learning a relation between an image including a content imageof an ear and an ear parameter of the ear.

(5)

The information processing device according to (4), wherein the firstlearning unit generates the ear parameter estimation model by learning arelation between the ear parameter and an ear image obtained byrendering three-dimensional data of the ear generated based on the earparameter.

(6)

The information processing device according to (5), wherein the firstlearning unit generates the ear parameter estimation model by learning arelation between a plurality of ear images obtained by changing textureof three-dimensional data of the ear or a head, a camera angle inrendering, or luminance in rendering and an ear parameter common to theear images.

(7)

The information processing device according to any one of (1) to (6),further comprising a second learning unit configured to generate thelearned model by learning a relation between an image including acontent image of an ear and a head-related transfer functioncorresponding to the ear.

(8)

The information processing device according to (7), wherein the secondlearning unit performs acoustic simulation for three-dimensional dataobtained by synthesizing three-dimensional data of the ear generatedbased on the ear parameter and three-dimensional data of a head, andgenerates the learned model by learning a relation between ahead-related transfer function obtained through the acoustic simulationand the ear parameter.

(9)

The information processing device according to (8), wherein the secondlearning unit compresses an information amount of the head-relatedtransfer function obtained through the acoustic simulation, andgenerates the learned model by learning a relation between thecompressed head-related transfer function and the ear parameter.

(10)

The information processing device according to (8) or (9), wherein thesecond learning unit sets a hearing point of three-dimensional data ofthe ear generated based on the ear parameter, and performs the acousticsimulation by using the set hearing point.

(11)

The information processing device according to any one of (1) to (10),further comprising a preprocessing unit configured to specify a contentimage of an ear of the user in a second image including a content imageof the entire head of the user, and detect a specified range as thefirst image, wherein the acquisition unit acquires the first imagedetected by the preprocessing unit.

(12)

The information processing device according to (11), wherein thepreprocessing unit specifies the range based on a relation between afeature point of the head of the user included in the second image and aposture of the user.

(13)

The information processing device according to (12), wherein when therange cannot be specified based on the relation between the featurepoint of the head of the user included in the second image and theposture of the user, the preprocessing unit newly requests acquisitionof an image different from the second image and including a contentimage of the entire head of the user.

(14)

The information processing device according to any one of (11) to (13),wherein the preprocessing unit specifies a content image of an ear ofthe user by correcting rotation of the second image based on correctioninformation included in the second image, and detects a specified rangeas the first image.

(15)

An information processing method by which a computer performs:

acquiring a first image including a content image of an ear of a user;and

calculating, based on the acquired first image, a head-related transferfunction corresponding to the user by using a learned model havinglearned to output a head-related transfer function corresponding to anear when an image including a content image of the ear is input.

(16)

An information processing program configured to cause a computer tofunction as:

an acquisition unit configured to acquire a first image including acontent image of an ear of a user; and

a calculation unit configured to calculate, based on the first imageacquired by the acquisition unit, a head-related transfer functioncorresponding to the user by using a learned model having learned tooutput a head-related transfer function corresponding to an ear when animage including a content image of the ear is input.

(17)

An information processing system including an information processingdevice and a user terminal, wherein

the user terminal includes

-   -   a preprocessing unit configured to specify a content image of an        ear of a user in a second image including a content image of the        entire head of the user and detect a specified range as a first        image, and    -   a transmission unit configured to transmit the first image        detected by the preprocessing unit to the information processing        device, and

the processing device includes

-   -   an acquisition unit configured to acquire the first image        including a content image of an ear of the user, and    -   a calculation unit configured to calculate, based on the first        image acquired by the acquisition unit, a head-related transfer        function corresponding to the user by using a learned model        having learned to output a head-related transfer function        corresponding to an ear when an image including a content image        of the ear is input.

REFERENCE SIGNS LIST

-   -   1 information processing system    -   10 user terminal    -   100 information processing device    -   110 communication unit    -   120 storage unit    -   130 control unit    -   131 learning unit    -   132 reception unit    -   133 ear model learning unit    -   134 image generation unit    -   135 ear parameter learning unit    -   136 HRTF learning unit    -   140 estimation unit    -   141 acquisition unit    -   142 calculation unit    -   143 provision unit    -   144 preprocessing unit    -   145A posture detection unit    -   145B ear detection unit    -   161 acquisition unit    -   162 preprocessing unit    -   163A posture detection unit    -   163B ear detection unit    -   164 transmission unit    -   165 reception unit

1. An information processing device comprising: an acquisition unitconfigured to acquire a first image including a content image of an earof a user; and a calculation unit configured to calculate, based on thefirst image acquired by the acquisition unit, a head-related transferfunction corresponding to the user by using a learned model havinglearned to output a head-related transfer function corresponding to anear when an image including a content image of the ear is input.
 2. Theinformation processing device according to claim 1, wherein theacquisition unit acquires an ear parameter that is a variablerepresenting a characteristic of the ear included in the first image,and the calculation unit calculates the head-related transfer functioncorresponding to the user by inputting the ear parameter to the learnedmodel.
 3. The information processing device according to claim 2,wherein the acquisition unit acquires the ear parameter of the earincluded in the first image by using an ear parameter estimation modelhaving learned to output an ear parameter corresponding to an ear whenan image including a content image of the ear is input.
 4. Theinformation processing device according to claim 3, further comprising afirst learning unit configured to generate the ear parameter estimationmodel by learning a relation between an image including a content imageof an ear and an ear parameter of the ear.
 5. The information processingdevice according to claim 4, wherein the first learning unit generatesthe ear parameter estimation model by learning a relation between theear parameter and an ear image obtained by rendering three-dimensionaldata of the ear generated based on the ear parameter.
 6. The informationprocessing device according to claim 5, wherein the first learning unitgenerates the ear parameter estimation model by learning a relationbetween a plurality of ear images obtained by changing texture ofthree-dimensional data of the ear or a head, a camera angle inrendering, or luminance in rendering and an ear parameter common to theear images.
 7. The information processing device according to claim 4,further comprising a second learning unit configured to generate thelearned model by learning a relation between an image including acontent image of an ear and a head-related transfer functioncorresponding to the ear.
 8. The information processing device accordingto claim 7, wherein the second learning unit performs acousticsimulation for three-dimensional data obtained by synthesizingthree-dimensional data of the ear generated based on the ear parameterand three-dimensional data of a head, and generates the learned model bylearning a relation between a head-related transfer function obtainedthrough the acoustic simulation and the ear parameter.
 9. Theinformation processing device according to claim 8, wherein the secondlearning unit compresses an information amount of the head-relatedtransfer function obtained through the acoustic simulation, andgenerates the learned model by learning a relation between thecompressed head-related transfer function and the ear parameter.
 10. Theinformation processing device according to claim 8, wherein the secondlearning unit sets a hearing point of three-dimensional data of the eargenerated based on the ear parameter, and performs the acousticsimulation by using the set hearing point.
 11. The informationprocessing device according to claim 1, further comprising apreprocessing unit configured to specify a content image of an ear ofthe user in a second image including a content image of the entire headof the user, and detect a specified range as the first image, whereinthe acquisition unit acquires the first image detected by thepreprocessing unit.
 12. The information processing device according toclaim 11, wherein the preprocessing unit specifies the range based on arelation between a feature point of the head of the user included in thesecond image and a posture of the user.
 13. The information processingdevice according to claim 12, wherein when the range cannot be specifiedbased on the relation between the feature point of the head of the userincluded in the second image and the posture of the user, thepreprocessing unit newly requests acquisition of an image different fromthe second image and including a content image of the entire head of theuser.
 14. The information processing device according to claim 11,wherein the preprocessing unit specifies a content image of an ear ofthe user by correcting rotation of the second image based on correctioninformation included in the second image, and detects a specified rangeas the first image.
 15. An information processing method by which acomputer performs: acquiring a first image including a content image ofan ear of a user; and calculating, based on the acquired first image, ahead-related transfer function corresponding to the user by using alearned model having learned to output a head-related transfer functioncorresponding to an ear when an image including a content image of theear is input.
 16. An information processing program configured to causea computer to function as: an acquisition unit configured to acquire afirst image including a content image of an ear of a user; and acalculation unit configured to calculate, based on the first imageacquired by the acquisition unit, a head-related transfer functioncorresponding to the user by using a learned model having learned tooutput a head-related transfer function corresponding to an ear when animage including a content image of the ear is input.